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Capacity Planning for the Masses — 
Using the SE Toolkit and Orca 


C apacity planning is often overlooked by systems adminis¬ 
trators until they need to justify additional hardware 
resources, and that’s the wrong time to start. In this article, 
Adrian Cockcroft (author of the Sun Performance and Tuning 
book) and l intend to show that starting a capacity plan within your 
Solaris environment is not as difficult as it sounds* We will 
introduce some key principles for beginning your capacity plan¬ 
ning and will show how we applied those principles and the 
tools presented in this article during the Winter 2002 Olympics. 


Capacity Planning Principles 

The fundamental principle behind any capacity plan begins with 
establishing an accurate baseline of your application's behavior 
over a period of time that can be easily interpreted. The following 
is a breakdown of this idea with the tools we use to achieve the 
fundamental principle. 

I. A capacity plan starts with accuracy, so use the right tools. 


Figure 1 Workload Example — Raw Data 
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Figure 2 Orca options 
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and allows long-term data views ranging from hours to years. Orca 
is also extended to use the SE Toolkit, and a new configuration file 
described below adds workollator.se workload data parsing and a 
set of standard plots. 

Now, let's apply these principles and tools to a real-world exam¬ 
ple. We used these tools to determine the capacity for the Winter 
2002 Olympics. To begin, we must clarify the location and I he 
purpose of each of the tools. They are deployed in the following 
manner: 

* SE Toolkit and workollator.se are installed and configured to run 
on the Solaris systems to be monitored. 

■ Orca, orca_gather.pL and workload_extract.sh are installed on 
the Solaris server, where information is to he centrally collected 
and processed. 

* The Web server is to be installed on the same server where the 
Orca html output is stored. It is only used to serve him I, and no 
CGE configuration is needed. 

Next, we can install the tools on the server to be monitored and 
start harvesting our application's stats. 

Download and install the SE Toolkit 3.3 from 
http: // WWW. setool kit. com. We will next configure a shell script 
to capture your system's workloads using workorallator.se on the 
host to be monitored. This script uses environment variables that 
identify the processes, which make the different workloads on the 
system. The environment variables are: 


PW_CMD_WK# — Filter based on the command name. 

PW_ARGS_WK# — Filter based on the argument strings. 
PW_USER_WK# — Fiber based on the user name. 

Listing 1 shows an actual example from SLC Sun servers to capture 
the workload information from the Winter 2002 Olympic Games. 
Listings for this article are available from the Sys Admin Web site at: 
http: //www.sysadmi nmag. coni. 

Note that the PW_CMDJ5 and PW_USER_3 on lines 5 and 6 
have the same workload number of _3. This is because we can 
stack variables and then filter. We are filtering all the Java 


Figure 5 Workollatorcfg — Workload search path 
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used to store data in units, RRD will generate time-defined graphs 
of the data stored within the given RRD. 

Ore a is managed and configured from a single configuration 
file, called orcallator.cfg (see Figure 2), The configuration tile is 
passed to the Ore a application, which provides site-specific infor¬ 
mation. such as where to go to find the files, the name of the files, 
and what type of graphs to plot. The command to run after you 
have configured Orca is normally : 

PJ orca -v -once *,/11 b/orcal 1 ator,cfg 

Figure 3 shows an example of the plot that is defined to generate 
the CPU usage of a process. 

We have provided a replacement for the orcallator.cfg, culled 
workollator.cfg, which should be passed to the Orca binary for the 
configuration information about how to generate the RRDs and 
graphs using the workload files. You must move the 
workollator.cfg tile to the same location as the orcallatorcfg, 
which is usually /usr/local/lib. Next, you must modify the workol¬ 
lator.cfg and give it the relevant parameters specific to your site. 

The following parameters tell Orca where to find or place 
RRDs and HTML tiles, as w ell as Orca's installation directory. Set 
these to the proper places. See Figure 4: 
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The format of each daily file should contain only the workload- 
specific data (see Figure 6). 

You should now 7 be able to run the Orca application and gener¬ 
ate your data. The - v flag will enable verbose logging on all of the 
processing, which is useful when running Orca, especially if it is 
the first time to run Orca: 


rrdjlir — Location in which RRD databases should be created 
or already exist, 

html di r — Location to write or update the html flics, 
base_d i r — Location of die Orca installation. 


./ORCA -v once ,,71 ib/warfcollater.cfg 

After the Orca application has completed and has returned you to 
the command prompt, navigate to the URL. which points to the 
Orca HTML files that have been generated. Once you navigate to 


The regular expression shown in Figure 5 
tells Orca where to find the files and the 
name of those files to be loaded into RRDs, 
Note that there is a bug in the current distri¬ 
bution of the workol laLor script, so you need 
k> remove ihe following line: 

<a href*"http://www, 5un.c ojh”> 

Processing the Workload Data 

Once all of the workload data has been 
transferred from the Solaris servers to the 
Orca server, w r e must parse each of the daily 
workload logs into individual files per work¬ 
load. The uorkload_extrctor.sh script 
(Listing 3) performs this task. Run this script 
in the directory containing all of the work¬ 
load files. The result should be that addi¬ 
tional directories are created (one for each 
hosts workload) under the OUTPUTDLR 
directory, which each contains log files for 
each day. 

Examples of the workload directories cre¬ 
ated on the system by the w ? orkload_ex trac¬ 
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The formula used in the workload-based capacity planning 
model to scale the workload capacity is: 


Total Sizing - Overhead + (Workload * Factor) 


Figure 8 Sample Daily Workloads (System 
Monti toring and Java Application) 
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Overhead is everything else that is not included in the workload 
being modeled or Overhead - (Total - Workload), Factor is how 
much the workload has changed by or scale. 




|f?rrj - WmM* Cfn up WDilfnlTjcf | w»tI sjvUIW 
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Amy Rich 


Questions and Answers 


Q l’m running Solaris 9 on a machine with one Ethernet card, 
but a large number of virtual interfaces. I want to use snoop to 
view the traffic on one of these virtual interfaces, but Fm not sure 
how to do that. The following command doesn’t seem to work: 

snoop -d hrneO: 1 

A Snoop is used to view packets on a physical interface, not 
logical ones. There is no way to look only at a virtual inter¬ 
face. If you need to snoop for traffic on this interface separately 
from the others, then I suggest adding another physical network 
card. If you’re looking for a particular kind of traffic, you can 
always try limiting the snoop output by using regular expressions. 

Q l'm trying to build a stripped-down Solaris 8 server to act as 
our gateway, l tried removing the following NFS packages, 
since this machine should never mount anything via NFS: 

SUNWnfscr Network File System (NFS) client support (Root) 

SUMWnfscu Network File System [NFS] client support (UsrJ 

SUNMnfscx Network File System (NFS) client support (Root) f64-bit) 

SUNWnfssr Network File System (NFS) server support (Root) 

SUNknfssu Network, Fite System [NFS) server support (Usr) 

SUNWnfssx Network File System (NFS) server support (Root) (64-bit) 

It turns out that SUNWvolu claims to depend on NFS, but I fig¬ 
ured that was just for using automounter, so I removed the NFS 
packages anyway. When I rebooted, void no longer worked, 
though, so obviously there's some deeper connection. Why on 
earth would vol d need NFS to mount things like CDROM s? 


require NFS at all, so all of the extraneous NFS client and server 
packages can be removed. 


A 


void treats everything like an NFS filesystem mounted under 
./vol. If you look in /etc/mnntab, you see the following line: 


host:vold(pid£07) /vol nfs ignore.dev—4540001 1054073390 

If VOl d mounts a CD ROM, then you’ll see an entry in /etc/mnntab 
for /vol/dev/dsk/cOt6dO/<something> as well (assuming your 
CDROM drive is at cUt6). The only time NFS is involved is during 
die actual mounting procedure. After that, the data is read by the 
kernel as HSFS or whatever formal exists on your media. 

When building secure machines, I always deinstall vol d as well: 


SUN'Wvolg 

SUNWvolr 

SUNWvolu 

SLiNWvolux 


Volume Management Graphical User Interface 
Volume Management* (Roof) 

Volume Management, (Usr) 

Volume Management (Usr) (64-bit) 


If necessary, you can always explicitly mount the CDROM by 
hand with the mount command. Doing the mount by hand doesn't 


q: 


Fm running Solaris 8. and I fear that I’ve unknowingly dam¬ 
aged my system. There have been no changes made recently 
that I know of, but this machine was shut down and then rebooted 
due to some maintenance that was being performed on our power 
systems at work. When the machine was turned back on, it started 
to boot, and then it displayed this message: 

mount:/usr Id,$0,1 not found 

I booted the machine from the network and then mounted /usr to 
took around. Id,so ] is in fact in /usr/Jib, and it looks fine. The 
checksums match other machines that are working ok, and the per* 
missions match as well Is something else broken, and the error 
message just misleading? Flave I been hacked and not realized it? 
I hope you can tell me that Fve somehow managed to break the 
machine myself (and there’s an easy fix like adding a package 
back in or something). Help! 

A You have the tool filesystem and the /usr filesystem ou two 
different slices, so it wouldn’t be looking at /usr/lib before 
/usr was mounted. Instead, the machine is looking in /etc/lib for 
Id,so. I. If you boot from the net or CDROM, and look in /etc/lib, 
you'll probably note that this file is missing. How that happened. 
I’m not sure (perhaps you removed it a while ago and the machine 
has not been rebooted since?), /eic/lib/ld.so, 1 is pan of SUNWcsr, 
as are /etc/lib/libdl,so. I and /eic/lib/nss_files,so.l. You may want 
to run pkgchk to see whether there are any other issues with 
SUNWcsr if you think you may have somehow damaged part of 
the package: 

pkgchk SUNWcsr 

The quick and dirty fix U> gel the machine up and running again is 
to copy /usr/Jib/ld.so, I over to /etc/lib. 

’m running Solaris 8 on an E450. We have a generic account 
^that should not be able to log in, run cron jobs, etc. My under¬ 
ling of the /etc/shadow file is that an entry’ with *NP* cannot 
log in but can still run cron jobs. An entry with *LK* should not be 
able to do either Fve modified the password entry to have *LK* in 
the password field, but the user can still run cron jobs. Is my under¬ 
standing flawed, or did I set something up w r mng or forget a step? 

A Your understanding is mostly correct, with one exception. 

An /ete/shadow entry with *NP* can actually log in if a 
password is not required $ la ssh aulhorized_keys, .rhosts, etc. 
The issue of your locked user still being able to run cron jobs 
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' "user @ local Without sticky host, the envelope would be changed 

to “user<§mailjiub*\ in order to protect against mailing loops. 

Since you’re not using MAIL_HUB, ruleset 5 never gets called, 
and that's where the LUSER_RELAY rules are (see the bottom 
two lines here): 



SLocal Jocaliddr 
SIocaladdr*5 
RS+ 

RI+ S| l#ok 
R$+ S| $#** 

R$+ 5| I* 


I; $L $| *>"Local 

s@ n 
#12 
t: SI 


localaddr" II 
no change 


# deal with plussed users so aliases work nicely 
R$+ + * $#1ocal If Uh I: II 

RS+ + I* $#1ocal If + $2 I; $1 + * 


# prepend an empty "forward host" on the front 

M+ I: <> $1 

# send unrecognized local users to a relay host 

^ >1+ S: < IL > Kuser II I) look up user 

R< I* > 1+ <> $; < > \i found; strip $L 

ff you remove the sticky host directive and regenerate your cf file, 
your LUSER_RELAY directive should work. 


Q l’m using the gcc-3.3 package from sunfreeware.com to 
bootstrap my own gcc-3.3 compiler on a Solaris 9 machine. 
The configure goes ok. and part of the build goes ok, but then it 
bombs out with the following errors: 

(cd Jibs && rm -f 11bstdc++Ja && In -s ../lihstdc++.la 
llbstdcH.la) 

Making ail in po 
Making all in testsuite 

mksh; Fatal error in reader: - missing from replacement macro reference 
Current working directory /usr/local/src/gcc-3,3/sparc-sun-solaris2.9/ \ 
sparcv9/libstdc++-v3/testsuite 
*** Error code 1 

make: Fatal error: Command failed for target ’all-recursive' 

Current working directory /usr/local/src/gcc-3.3/sparc-sun-soUMs2,9/ \ 
sparcv9/Iibstdc++-v3 
*** Error code 1 

make: Fatal error: Command failed for target ’al1-recursive-am' 

Current working directory /usr/local/src/gcc-3.3/sparc'Sun‘Solaris2,9/ \ 
sparcv9/libstdc++*v3 

*** f fr{ j r C0( j e i 

make: Fatal error: Command failed for target "multi-do 1 

Current working directory /usr71ocal/src/gcc-3.3/sparc-sun“SOlaris2,9/ \ 

1fbstdc++-v3 
*** Error code 1 

make: Fatal error: Command failed for target ‘all-multi’ 


Current working directory /usr/local/src/gcc-3*3/sparc-sun-solaris2.9/ \ 
Ubstdc++-v3 
*** Error code 1 

make: Fatal error: Command failed for target 'all-recurs!ve-am f 
Current working directory /usr/local/src/gcc-3,3/sparc sun solaris2.9/ \ 
11bstdc++-v3 
*** Error code 1 

make: Fatal error: Command failed for target ’all-target-1ibstdc++*v3' 
Current working directory /usr/local/src/gcc-3,3 
*** Error code 1 

make: Fatal error: Command failed for target 'bootstrap' 

I’m not sure what mksh is, or what replacement macro reference 
it's talking about. Obviously, someone has gotten the compile to 
work because there's a package already, but I can’t seem to work 
around this issue. Do you have any suggestions? 

A You get this error when you’re trying to compile some of the 
extra languages that come with gcc but you aren't using GNU 
make. You can either specify that you only want C, or you can 
install GNU make (or put it first in your path) and reconfigure gcc. 
I would suggest the latter, since various other pieces of software 
will also want GNU make, 

Amy Rich, president of the Boston based Ocean wave Consulting, Inc. 
(http:/1 WWW. OCefinwd vb, com), has been a UNIX systems administrator for more than 
JO years. She received a BSCS at Worcester Polytechnic Institute, and can be reached 
at: qndioceanwave. com. 
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E-mail text forms as PDF 

Form2PDF is a standalone program 
that converts text files to PDF adding a 
background image in the process. The 
resulting PDF document can be sent as 
e-mail using the built-in SMTP client 


Form 2 PD F is available for a variety of 
UNIX and Linux platforms. 


VI s It www. i nfof I ex c on n ect. se/f o rm 2 pdf 
or call +46-8-555 768 60 for more 
information. 


Solaris Administration Supplement 2003 


wwwsysadminmag.com 


Sys Admin — 11 



































Datasheet The Solaris ’ 9 Operating System 


The best gets better. The Solaris 9 Operating System provides 
a foundation for Network Identity Management through 
integration of the Sun'" ONE Directory Server and the Java 2 
Platform, Enterprise Edition (J2EE) technology-based Sun 
ONE Application Server. 


performance and sea Lability of the Solaris 9 OS, 
customers can immediately see an improvement 
In the price/performance ratio of their servers, 
and thus, increase the return on their investments. 
The Solaris 9 Operating System offers the 
capabilities to scale and support massive systems 
and applications: 

* One million simultaneous processes on 
a single system 

* Up to 128 CPUs in a single system and 
848 CPUs in a clustered environment 

* Support for up to 576 GB of memory 

* More than four billion network connections 

* Up to 252 TB file systems with Sun 
StorEdge" QF5 

* IPv 6 , enabling a 128-bit IP address space 
■ A 64-bit lava virtual machine 

In addition, Solaris 9 software introduces 
a group of new features designed to increase 
system performance and scalability. 

Solaris 9 Threads Model 
By automatically utilizing an enhanced multi¬ 
threading library, applications on the Solaris 9 
platform demonstrate greater scalability and an 
overalL performance increase of up to 4 times. 
There are no changes to the interfaces and no 
need to recode applications. 

Solaris 9 Memory Optimizer 

* Multiple Page Size Support. Applications 
can use multiple page sizes ranging from 8 
KB to 4 MB for different memory segments. 
This increases performance by enhancing 


resource efficiency and reducing overhead. 
The binary is left untouched, and page 
sizes are dynamically changed as an appli¬ 
cation executes. 

■ Advanced Page Coloring. The Solaris 9 
Operating System includes enhancements 
to the algorithm that control virtual/physi- 
cai pages and how they are reached. As a 
result, system performance is increased for 
particularly heavy user loads. 

* Memory Placement Optimization. Solaris 9 
software is designed to optimize memory 
management in the way that best suits the 
particular servers on which it is running. 

* Memory Allocation. Solaris now allocates 
memory to user-level applications much 
faster, and enables users to more easily 
debug memory leaks in their applications. 

UNIX File System (UFS) Enhancements 

* UFS Performance Enhancements. For 
databases created on a UFS file system, 

UFS Concurrent Direct 1/0 provides near raw 
device performance, leading to an 87-percent 
improvement in TPC-C measurements on 
OLTP workloads. UFS with logging enabled 
ensures fast file system recovery and outper¬ 
forms non-logging UFS. Administrators can 
now get the benefits of logging with UFS 
without having to worry about the effect on 
performance. The UFS file system is also 
compatible with the standard EFI disk label, 

* Multiterabyte UFS. For 64-bit SPARC plat¬ 
forms, UFS now supports much Larger file 
systems, up to 16 TB in size. 

* mkfs. Enhancements to the mkfs command 
have dramatically reduced the time it takes 
to create a UNIX file system; you can now 
create a UNIX file system 96 times faster. 

* 64-Bit lava Virtual Machine. With ]ava 2 
Platform, Standard Edition (| 2 SE“) 1.4 soft¬ 
ware, users benefit from full 64-bit support 
in the Java Hotspot" virtual machine (VM). 
This feature, combined with the Java HotSpot 
Server VM code optimizer, has significantly 
improved java Servlet performance. 
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The first implementation of Solaris 
Containers is Solaris 9 Resource Manager 
software, which is integrated into the SoLaris 9 
Operating System. It consists of a set of resource 
management and network quality-of-service 
features. Solaris 9 Resource Manager enables 
administrators to alLotate system and network 
resources to multiple users, groups, or applica¬ 
tions to provide more predictable service levels. 
Customers can set and enforce policies that 
control and monitor how resources are used, 
and generate extended accounting informa¬ 
tion for billing or chargeback purposes. Solaris 
9 Resource Manager software redefines the 
traditional single application system model 
and offers a better solution by enabling server 
consolidation to reduce service costs while 
delivering more predictable service levels. 

Data Management 

In the Solaris 9 platform, new features are 
introduced to help manage the data that gives 
a company its competitive advantage. 

* Solaris Volume Manager. The Solaris Volume 
Manager storage management tool, which 
now includes a GUI, is integrated into the 
Solaris 9 Operating System. It enables users 
to manage large numbers of disks into log ical 
volumes. The Solaris Volume Manager has 
been enhanced to support disks larger than 

l TB in size. Solaris Live Upgrade makes it 
much easier for customers to adopt Solaris 
Volume Manager because Solaris Live 
Upgrade greatly simplifies migration from 
other volume managers to Solaris Volume 
Manager. Reconfiguration Coordination 
Manager (RCM) support is integrated in 
Solaris Volume Manager, allowing the safe 
removal of disks through Dynamic Reconfig¬ 
uration (DR) requests. 

* Soft Disk Partitions. The soft disk partitions 
feature increases the number of file systems 
per device from eight to thousands. Parti¬ 
tions can also be created on top of previously 
defined logical volumes, giving administra¬ 


tors additional flexibility in configuring and 
managing the volumes. 

* UFS Snapshot. With UFS snapshot, the 
Solaris 9 Operating System provides an online 
backup mechanism by creating a point-in-time 
image of the file systems. It helps eliminate 
downtime or offline time previously required 
to guarantee a consistent backup. 

System Management 

* Administration. The Solaris 9 Operating 
System provides a wide range of administra¬ 
tion tools that assist both user and system 
administration tasks. It provides command¬ 
line tools and GUIs for managing users, 
resources, and disk storage. 

Solaris Patch Manager offers the most 
comprehensive patch management features 
for the Solaris Operating System, Admini¬ 
strators now can analyze the patch state of 
a system and automatically download the 
recommended patches. They are provided 
with the install order necessary to accom¬ 
modate patch dependencies, and can use 
the tools on local and remote systems. Ail 
patches delivered via Solaris Patch Manager 
are digitally signed, helping ensure that the 
patches are from Sun and have not been 
altered in transmission. 

* Monitoring and Management Sun Manage¬ 
ment Center provides a powerful, easy-to-use 
single management point for all Sun servers 
and storage, independent of geographic 
location. System administrators can perform 
re mot e sy st e m co nf i g u rati 0 n f pe rfo rma n ce 
monitoring, and isolate hardware and soft¬ 
ware faults through a singLe interface. Sun 
Management Center easily integrates with 
enterprise management frameworks, and 
provides a central facility for managing 
events and alarms, automated responses, 
and diagnostics. 


Soio/vs Provisioning Services 

- Solaris Web Start. Solaris Web Start software 
simplifies the installation, setup, and admini¬ 
stration of applications written for both 
Solaris and Java technology-based environ¬ 
ments with point-and-click ease of use. 

* Solaris Flash and Solaris Live Upgrade. 
Solaris Flash makes it easy to provision large 
numbers of servers having similar configura¬ 
tions. ft enables administrators to create a 
single reference installation of the entire 
system software stack and replicate the 
installation on a numbers of servers. Solaris 
Flash reduces installation time and configu¬ 
ration complexity, and simplifies the process 
of redeployment to support different service 
levels as customer demands change, 

Solaris Live Upgrade Is integrated to 
work with Solaris Flash technology. This 
functionality provides a mechanism to install 
a Solaris Flash archive on an inactive boot 
environment while the active boot environ¬ 
ment is fully functional. A simple reboot will 
migrate the system to the updated environ¬ 
ment when the process is finished. This 
significantly reduces the installation time. 

* Solaris jumpStart Software. With SoLaris 
)umpStart software, the Solaris Operating 
Sys-tem and applications placed on a central 
server can be used to remotely set up a 
Solaris system anywhere on the network. 

* Secure WAN Boot. Sun's new secure WAN 
boot technology provides mechanisms that 
enable system administrators to boot and 
instaLl new or upgrade systems over a wide 
area network. It further enhances system 
scalability by enabling administrators to 
remotely install multiple duplicate systems, 
such as Web servers or application servers, 
over geographically dispersed areas, 

» Sun Management Center Change Manager, 
Available separately, Sun Management Cen¬ 
ter Change Manager extends Solaris platform 
functionality by offering advanced provision¬ 
ing capabilities. It delivers a fast and easy 
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On the Web sun. com /Solaris 


The Solaris 9 Operating System 


Linux Compatibility 

in today's world of heterogeneous computing, 
compatibility leads to efficiency. Combining 
the Linux community with thousands of Solaris 
software developers and nearly three million 
lava and XML software developers, Sun provides 
customers with unified access to the broadest 
array of innovation in the industry on which 
to provide services, tn the Solaris 9 Operating 
System, more Linux applications, tools, and 
APIs are made available. 

* API Compatibility. Common libraries and buiId 
environments, such as libxml, glib, and GTK+* 
are integrated in the Solaris Operating Environ¬ 
ment to streamline source code development 
across Linux and Solaris environments. 

* Application Compatibility. Common Linux 
applications, such as Samba, Apache, Linux 
(GNU) commands, etc., are included in the 
Solaris 9 Operating System.The Solaris Software 
Companion CD has an even more comprehen¬ 
sive set of free software. 

Now Available for xSB-based Platforms 

Sun extends Solaris software's value to xS 6 
systems by providing the same benefits and 
features as available on SPARC® systems. 
Therefore, the Solaris OS x 86 Platform Edition 
is well-suited for companies that have already 
invested in Solaris software expertise and 
commercial Solaris applications; that need a 
predictable release and update process with 
an application compatibility guarantee; and 
that want Solaris source code access — all 
while still leveraging standard x 86 *based 


system hardware and common management 
tools. Applications that need even higher 
levels of security than those found in the 
Solaris OS can also take advantage of Trusted 
Solaris software on x 8 G-based systems. 

Modern Desktop 

The GNOME 2,0 Desktop takes desktop com¬ 
puting on the Solaris Operating System to an 
entirely new level. It not only provides an easy- 
to-use, appealing interface, but also a wealth 
of productivity tools and utilities to help users 
get their work done efficiently. 

Standards 

* Interface Standards: X/Open* UNIX 9& _ 

* Graphic Standards: X11, PostScript", Display PostScript*. OpenGL® 

- Desktop Standards: CDE (Common Desktop Erwironment), 

GNOME, Motif _ 

* Object Standards: java IDL _ 

1 Connectivity Standards: GNC, ONC+", NFS, Web NFS", SMB 
technologies, XML 

* Internet Standards: HTTP, FTP, Telnet, DNS, NTR IMAP4, DHCP, 

SNMp IPv6, IPBec, Kerberos, SASL, OCT ___ 

* Protocols: LDAP V 3 IETF, RfCs 1323,1510, mi, 1869,1870, 

1 B 91 - 1994 , 19 &Sh 1996 * 20lB r 2136, 2045 , 2078 _ 

* Web Services Standards: Java API for XML (JAXR), ebXML, UDDt 

- XML Messaging: Java API for XML Messaging flAXM} - SOAP. 

ebXML, Tfi&p SOAP-RP; \m API for XML-based RPC (|AX-RPC) - 
SOAP. WSB _ __ 

- XML Processing: Java API lot XML Processing (JAXP) — XSLT, 
$AX2 f D0M2. Schema, XSLTt 


System Requirements 

■ SPARC® 32-bit and 64-bit platforms 

* *86 (32-bit) Sun and third-party platforms J 5 ee. 5 un.c 0 m/ 
bigadmin/fid for a hardware compatibility list) 

- Disk Space: Goo MB for desktop systems; 1 GB for servers 

* Memory: 64 MB minimum for *86 platforms; 128 MB mini mum 

for SPARC platforms 


learn More 

Get the inside story on the trends and technologies 
shaping the future of computing by signing up for the 
Sun Inner Circle program. You'll receive a monthly 
newsletter packed with information on the latest 
innovations, plus access to a wealth of resources. 
Register today to join the Sun Inner Circle Program 
at sun.com/foinic. 

To receive additional information on Sun software, 
products, programs, and solutions, visit sun.com/ 
software. 

For More Information 

To Learn more about the Solaris 9 Operating 
System, visit sun.com/solaris. 

For more information on Solaris OS x 8 G 
Platform Edition, please visit sun.com/solaris/x 86 
For additional technical and development 
information, please visit the BigAdmirf portal 
at sun.com/bigadmin. 



So v,,. lewitfe Sates Offices; * * iN.r ch. WtsLand Gftiarat) -33-J3-D&7-4680, Argentina 154114317-5Gao, AustraliaAustr-a **3*60563-0, Belgium -37-3-704*8000, Bra.r.t -95 ls-yiar-rm C anada t-jJOS-47V- 

1 • i .‘1734500, Col 1 -1-1* +5?*-pJW333, CainiftOJ'WMkh intteofiildttrt: States + ’ 503-495-8411. Cz8ch‘ReCa^ic t-tiTO-7-3300-931 j, Denmark -45 4556 ROOD. Ft.yrt +202570-34*17, Estonia «-372'G-303-iiCM, r.inlwrf 
. . sti. France -*3 l-ia •: :-orc—CC; Germany d, Ctee-tf -j.n-s-R'if-gj r*. Hun§s&y *56-1 489-8900, Iceland +i54-563-3o:o, lnd»a--B*rK|atorf 7255450: Ntv, 3e:m rgijliulOMcOl iSnifiiRa *31- 

2? -i • |relent • -C-i. ii afil •►977-9-97105CO, Italy ni^-OJ-64151 J, |ac-:m *8i-5;5if.7-5OO0, KaAikLsiun *7 5274 466/"*1, W»9 *832-2193:5114. Latvia •♦srl-’/Sfl-J'TW Litbunnut-370-7;9-846W, lUJtfimuourg *352 -1 -5 

it ,3 Malaysia +fic•1-/1161/1 Li, Mexico *5; 5-258-6100,the NKhtt-und-r -*0Hj*3X-33-45-J-OOP. Newfra'laitd -AuiHand -154-9 3?6*6fiiOO*,. 1 -64-4-46i-tJ78o. , Ni2iwaM »47’2J 369500. Fwple’i Jlajftwlu: of Lh.i in-Beil mu 
ifc i ;ii>3-5588,Chetrtdu *8. .8*6«- >333; Guangzfic-ufifero-B755*56O0] Stwr-c-ba- .86*2i-ii46ti'i23fl,Non<r.Kt*n4-n»32-2?02-!iij8B. Poland >4R-2?.-8747S0Q, Perineal +iw-2:-4:'3*tOCO. Russia *7-507*935*841 

ti* 1 Slovak Republic -*-4 434* $4**, South Africa *7711 356-63 Op, Spain 1-34-91 59^9900. Sweden *#4431-10-00. SAliWflriand-Gcmun 4W*908-90-(i0: frencti 4i-22-.999-<M44. fawa* +3B6'3 B? J?*99. :3. Thai land 

. J &BSB, Turku/ *90214 . \-2>to, Uri-recf Arab'Emirates ^7U-33|W83* United King-deirr United or *i-$60-96o-43M, IteneeuaJ* -38-2-906*3800. 

r • ■ 1 * ■£ 3003 Sun Wfewsvitems Inc, ALL rights reserved. sen, Sun Microsystems, the fti« laqcr. tfgAPtun, Sava, |ava vlo'-Spoi, (2rF, |2Sf, Ni. ONC. ONC-. Sotefii. Solaris jumpStar Soiam |mernrhieAuinentitatfon 
Met lie*3d 1 • , :II, jcreen, Siin StiffMge. Trusted Solaris, and WebKFi an! tr.wtemarttor reqriwrt-d trade-marts ol Sun M.nn«.ys:e-ns, int. in Hie Umte.d States on a ott'ei counting mi, l-FAF c ti:n1en-a-ks 
gi . rnder cense and m: -a,fm arks m registered Trademarks of SMRC Inrsrnfttmnal. Inc. In. ttie U.S. and Other <ouo:ri«5, Products-tiearing SPARC ti*dr»Ria.»iw arc nased-oopn an architectnre dWWtt08n W San 
,V, 1 .rams if»c 1 .p.jsiS-'ripr £hmi *v Post5c«pt are tradio.=TOrfcs or r?gist«reil trddfnMiri i of Adobe SustKjis, intctfw rated, whkn may he wiSt.efe^ In terrain [Miirtkrioris pner-ui >i.a -registereri trsrderrwix Olslrog© 
A 'nc \im r, a reolsuo-r traih^aik in the United itAtbs and oilier countries, «rtti»h*ly lieenw-d ItirauyK X'-'Open Co-ir^any. Ltd. X'^'pen.^a regiswred tr*il«nnwk n-i X/Opeit CtmipOnv, Ltd. loftmtifttan W ^ 
cl '1,1 jfhoutnoMK ' Printed,nUSA^n3m.b&4-3 


Vendor Sponsored Content — C3 
























7/ Applications Actions 


Elte view Terminal Qo tEeJp 

/* Streams Module Linkage */ 

static struct nadlintoge modlinkage - { tfoDREt 

/* Solans Driver Interface */ 
int _init(void) { 

return mod_installfSaiodliTikage) ; 


Fly guy 006 : 
SURFDUDE 
Flyguy 006 : 
SURFDUDE 


WAY! 

NO WAY! 
WAY! 

NO WAAY! 


int fxni (void) ( 

return *od_re»cve (Smodlinkage) 


_info(Struct modinfe *mcdinfop) { 
return mod^info(S®odlinkaga, aodinfop) 


Instant Message 


A serious notebook for Solaris users. 


tf you re thinking about buying a notebook, why settle for a Windows' 5 laptop when you can have 
Solaris® to go? Because that's what a SPARCLE* notebook from Tadpole® gives you With its 
sleek design, 2GB memory and WiFi, SPARCTE looks like a Windows notebook. The difference? 
it runs on SPARC® meaning all the code you generate is 100% compatible across every machine 
Sun makes. So now you'll be able to develop, sell, support, and use Solaris anywhere. WAY. 

Enter to win a SPARCTE notebook, www.tadpolecomputer.com/serious 


♦a 


Star Off ice' 


ULTRASPARC 

PfllUfh jm 



You run Solaris. So why 
does your notebook run 
Windows? Our sentiments 
exactly. Well, now there's 
SPAR OLE from Tadpole. 
Starting at just $2,995. 


* 2003 Tijd P°!e Computer, Inc SPARC EE and Tadpole are registered 


trademarks of Tadpole Computer. Jnc. AN Other trademarks are property ef their respective t 
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way to mstaU, upgrade, and audit the soft¬ 
ware on your systems- Change Manager 
enables today's business to quickly and easily 
provision software stacks to their servers by 
providing automated tools for installation or 
upgrade of hundreds of servers at a time - 
while the systems continue to operate. This 
can save IT costs and minimize disruptions 
of services. Sun Management Center Change 
Manager has an easy-to-use Web browser 
interface as well as a command-line interface 
fCLIJ for experienced system administrators. 

* User Management, The Sun ONE Directory 
Server is integrated into the Solaris 9 
Operating System. It creates an LDAP-based 
environment that can scale to millions of 
users. An NlS+ to LDAP transition kit is avail¬ 
able to help ease migration from NIS+ to 
LDAP. Also, the secure LDAP client has been 
enhanced to support various encryption 
mechanisms, including DfGEST-MDs and SSL, 
enabling secure password management 
through the Sun ONE Directory Server, 

The Solaris 9 os Js Secure 

The Solaris Operating System has been built 
to securely interconnect with other systems 
and be more secure from the viruses and worms 
that plague software designed without security 
and networking in mind. These aspects of net¬ 
work and platform security are critical — 
whether building an enterprise network or 
providing services to millions of users over the 
internet. The Solaris 9 OS includes a number 
of new security features. 

Network S ecunty 

The network security provided by the Solaris 9 
Operating System helps ensure secure authen¬ 
tication over the network, secure remote 
access, secure network connections that can't 
be snooped, and protection from network 
based attacks. 


* Solans Secure Shell. Solaris Secure Shell 
software enables strong authentication of 
both the client and server machines as well 
as users for use m remote access solutions. 
It also provides encryption for privacy using 
the 3DES, AES and Blowfish algorithms, ft is 
compatible with otherSecure Shell protocol 
implementations. 

* IPSec With Internet Key Exchange 
(IPSec/lKE). The IPSec standard provides 
data integrity, privacy, and authentication 
of network traffic between servers for both 
IPv4 and IPv6 networks, IKE provides a 
standards-based mechanism for exchange 
of encryption keying material and digital 
certificates for use on IPv4 and IPv6 networks. 
Encryption is transparent to applications, and 
as such requires no changes to the customer 
experience. fPSec/IKE features DES, 3 DES, 

AES (256-bit), and Blowfish (443-bit) encryp¬ 
tion — all approved for export and use 
worldwide — and is compatible with other 
fPSec/IKE implementations. 

* SunScreen" 3.2, Software, SunScreen 3.2 
software is a high-speed, stateful packet- 
filtering firewall that offers advanced features 
that protect a single system or an entire net¬ 
work of servers, it is now included with the 
Solaris 9 Operating System at no extra charge. 
Kerberos Single Sign-On Environment, The 
Solaris Enterprise Authentication Mechanism" 
server and client is included in the Solaris 9 
Operating System to provide single sign 
on capabilities for servers and applications. 
Kerberos-enabled versions of telnet, r* 
commands and more are available as a free 
download from the Sun Download Center, 

•TCP Wrappers. Based on open source, TCP 
Wrappers provides a means of protecting 
your server from incoming traffic. Connec¬ 
tions can be limited by DNS domains, IP 
addresses, or by substituting wild cards for 
part of the domains or addresses. 


Platform Security 

The platform security features in the Solaris 9 
Operating System support server and cluster 
hardening through; 

* Pluggable authentication modules (PAM): 
flexible security standards 

* SSL-encrypted LDAP authentication for 
native login 

* Role-based access control (RBAC) 

* Kernel Pseudo Random Number Generator 

* Nonexecutable program stacks 

* Modular software packaging 

* Extensive higher performance auditing 
with KML output 

Pluggable algorithms for password encryp¬ 
tion (Crypt, MDs, and Blowfish included) 
Smart card authentication support 

* Secure by default file permissions 

* Free Solaris Fingerprint Database for file 
integrity verification 

* Free Solaris Security Toolkit for proven 
system hardening 

Compatibility 

Compatibility is one of the hallmarks of the 
Solaris Operating System, and is key to enabling 
customers to move up the hardware product 
line without having to port or recompile their 
applications. Solaris software supports a pub¬ 
lic application binary interface (AB!) which 
guarantees that conforming applications will 
run on all Sun servers without modification. 

SolCAT: Solaris Compatibility Assurance Toolkit 
The Solaris Compatibility Assurance Toolkit 
(SolCAT) is a collection of tools and services, 
including the Sun Guarantee Test Suite and the 
Certification Test Suite, to help customers and 
ISVs easily and seamlessly ensure that their 
applications will run on the latest version of 
Solaris software. 
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The Solaris 9 OS fs Available 

With businesses operating around the clock 
and around the globe, organizations no longer 
know when their customers might demand 
their services. The reliability of the Solaris 9 
Operating System increases users' confidence 
that their long-running and resource-intensive 
applications will execute without interruption. 

The Solaris 9 Operating System is designed 
with a small, compact kernel that limits the 
exposure to errors that can crash a system. It 
is also designed with a clear distinction between 
the kernel, shared libraries, and applications to 
further limit the impact of application failure. 

To deliver mainframe reliability with Internet 
agility, the Solaris 9 OS raises the bar for relia¬ 
bility, availability, and serviceability (RASJ by 
strengthening the following characteristics in 
every aspect of the development process. 

Robust 

• Dynamic System Domains. The Solaris 9 
Operating System provides failure contain¬ 
ment and high-level control over system 
resource allocations in electronically isolated 
partitions. Partition boundaries can be ad¬ 
justed on the fly to rapidly adapt to changing 
workloads, or on a scheduled basis to enable 
resource shift between applications. This 
helps improve service levels economically. 

Solaris Containers. Solaris Containers 
isolate software applications or services 
using flexible, software-defined boundaries. 

If a fault occurs in a user-level process, the 
container boundary would then prevent 
propagation of the failure to other containers, 
Kernel and User Mode Separation. With the 
kernel occupying a protected address space, 
and the user-level libraries and applications 
occupying separate user address spaces, it is 
extremely difficult for a user error to cause a 
system failure. This significantly enhances 
system availability. 


* Kernel and Device Driver Hardening. The 

kernel and device drivers are further hardened 
by identifying and eliminating panics and 
kernel memory leaks. The new device driver 
testing framework enables developers to 
stress drivers and simulate hardware failures 
to further validate that the drivers can han¬ 
dle u nforese ea bie c i rc u m sta n c e s. 

Recoverable 

* File System journaling. By reducing file sys¬ 
tem check times during reboot, the Solaris 9 
Operating System increases recoverability. 

- NFS Failover With NFS Failover, client systems 
can retry network file access on an alternate 
server when the primary server fails. 

- Network Multipathing. The Network Multi- 
pathing feature allows multiple network con¬ 
nections to provide load spreading and failover, 

• Sun StorEdge Traffic Manager. The Solaris 9 
Operating System supports multiple paths 
for I/O devices, such as SCSI- and Fiber 
Channel-accessible storage, for load balancing 
and failover. 

Mo nogeo bie 

Dynamic Reconfiguration. Dynamic Reconfig¬ 
uration enables system configurations to be 
changed without rebooting, minimizing 
planned and unplanned downtime. 

- Solaris Live Upgrade. The Solaris Operating 
System can be upgraded while the system 
stilt running, significantly reducing the 
usual service outage time associated with 
standard upgrades. Enhancements to the 
Solaris Live Upgrade Boot Environment 
allow Solaris JumpStarT software to auto¬ 
matically create boot environments during 
Install time, resulting in a faster Solaris 
Live Upgrade copy process. 


Observable 

* Modular Debugger Framework. The Modular 
Debugger Framework is an extensible utility 
that enables low-level debugging and editing 
of the live OS. 

* Kernel Analysis Tools. The Solaris 9 Oper¬ 
ating System provides a comprehensive set 
of "on-the-fly” kernel analysis tools, kstat, 
lockstat, prstat, and cpustat provide statis¬ 
tics and parameters for the kernel, file locks, 
process status, and CPU utilization respec¬ 
tively. truss can be used to wrap an applica¬ 
tion so that ail of its system calls can be 
observed externally. 

Availability With Sun Cluster 3.0 
Sun Cluster software, available separately, 
provides continuous access to services via Global 
Network and File Services. Data, networks, and 
devices are available to all domains in the 
SunPiex systems as well as to applications 
running on ary domain. 

The Solaris 9 OS Is Manageable 

As IT infrastructures grow increasingly larger 
and more complex, IT organizations must 
optimize for efficiency. Deploying and redeploy- 
mg servers is a daily event. To stay ahead, busi¬ 
nesses today must manage IT growth, while at 
the same time reducing complexity and man¬ 
agement costs. The Solaris 9 Operating System 
provides a rich set of management facilities 
that can simplify the process of securely instal¬ 
ling and deploying the software stack, resulting 
(n lower cost of operation. 

Solaris Containers and Resource Management 
Solaris Containers create an execution environ¬ 
ment within a single instance of the Solaris 
Operating System, and provide full resource 
containment, fault isolation, and security 
isolation. This common approach simplifies 
service provisioning and makes it easier to 
consolidate applications onto fewer servers 
without concern about resource constraints, 
fault propagation, or security. 


Vendor Sponsored Content 


Datasheet The Solaris' 9 Operating System 


On the Web sun.com/solaris 


The Solaris” 9 Operating System 


Increase service levels. Decrease costs. Reduce risks. 



Key feature highlights 

-Integration of the Sun" ONE Directory Server 
and the ]2EE" Sun ONE Application Server in 
the Solaris' 9 Operating System provides the 
foundation for Network Identity Management 

- Delivers the performance and stability to meet 
production database and file system require' 
ments — with no incremental costs. 

- Provisioning and change management 
provides secure installation and deployment 
of software stacks. 

-Solaris Containers, combined with Dynamic 
System Domains, enable customers to get 
higher utilization of their system resources, 

- Provides out-of-the-box security solutions and 
a fully integrated suite of security services to 
deliver the highest Levels of security. 
Champions the RA 5 Lifestyle into every step 
of software development process. 

-Configuration services and Patch Manager, 
combined with SunPLex" systems, deliver 
high service levels, 

-Applications run faster without recompiling 
and recoding. 

-Compatibility with previous versions 
and built-in Linux compatibility offer 
investment protection. 

-The Solaris OS has been extended to x86 
servers, giving customers the freedom to 
choose the solution that best meets their 
business needs. 


Today, businesses are rethinking how they create, manage, extend, and ultimately deliver 
information technology (IT) services with greater functionality and reduced cost and complexity. 
Managing data center complexity from a services perspective lets businesses focus on choosing 
the right soLution for the fob at hand, rather than managing individual systems. 

Since its inception in 19S2, Sun's vision and strategy has been the same: connect every¬ 
thing through network computing. Sun continues to leverage open standards and technologies, 
innovate on top of them, and create the types of systems customers demand. The foundations 
for this revolution are Sun's lava ' Enterprise System software, |ava technology, Nl’ software, 
and the Solaris" Operating System, Providing the most scalable product line packed with fea¬ 
tures, tighter integration, more complete testing, and the highest security levels for general 
purpose servers. Sun gives companies the freedom to choose the systems and software that 
best meet their business needs. 


For more than ten years, the Solaris Oper¬ 
ating System {OS) has delivered the power, 
massive scalability, high levels of security, and 
mainframe-class functionality that companies 
demand. It's the leading UNIX* 6 * environment 
- and the choice for powering enterprise 
networks that need to deliver information to 
networked users at any time, any place, on 
any platform. 


Sun is now bringing the industrial strength 
of the Solaris OS and the military-grade security 
of the Trusted Solaris ' OS to x£J6-based systems. 
Providing the same functionality through 
the same source code, the Solaris and Solaris 
x86 environments deliver stability and open 
source innovation on both UltraSPARC® and 
xS6-based systems. 



The Solaris 9 OS Is Scalable 

Companies offering Web-based services can no 
longer predict how many customers might visit 
their sites. Today's companies need the ability 
to grow to support millions of new customers 
overnight. 

The Solaris 9 Operating System is the third 
major release of the complete 64-bit computing 
environment tuned for Sun's powerful line of 
highly scalable 64-bit servers. The multithreaded, 
fully preemptible kernel delivers much faster 
performance for core system functions and enter¬ 
prise applications. And with the new increased 
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was actually a bug that was recently fixed, though. Ironically, 
because of the fix, various people have tiled bug reports with 
Sunsolve saying that w hat was previously functioning has mys¬ 
teriously stopped working, A couple reports state that the “fix” 
happened in 108993-18, so I'd try installing the latest 108993 
patch and see if that helps, 

Q This is sort of an odd request, but is necessitated by some test¬ 
ing that our development group would like to do. We're run¬ 
ning Solaris 8 on a number of different UltraSPARC machines. We 
need make time appear to go faster or slower on these machines 
depending on the test. We don’t need to muck with the hardware 
clock Just have the time/date change according to the rate we want. 
With all of the Y2K testing that had to happen several years ago, I 
presume someone must have come up with a product that does this. 
It vvould be great if you could point me at such a beast. 

A If you’re only concerned with the time and date as reported by 
the 05, then you need look no farther than your base Solaris 
install. The Aisr/b in/d ate command takes an optional flag of -a to 
skew the system clock one way or another From the man page: 

-a [-]sssdff Slowly adjust the time by sss.fff seconds (fff represents 
fractions of a second). This adjustment can be positive or negative. 

The system's clock will be sped up or slowed down until it has 
drifted by the number of seconds specified. Only the superuser may 
adjust the time, 

Q W r e’re running Solaris 9 with a bunch of third party software 
installed. Unfortunately, this means we have overlapping man 
pages in /usr/share and /usr/local Our standard system path has 
/usr/local before /usr/bin, for example, so most of the time people 
want the corresponding man pages in /usr/local. Occasionally, 
especially when writing shell scripts and explicitly using binaries 
in /usr/bin, people will w r ant Lhc man page for the stock Solaris 
command, 

A good example is the df command. Because the MAN PATH 
has /usr/local/man before /usr/sbare/man, people who run man df 
always get the FSF man page. Just calling nroff with the path of 
the system man page doesn't work because Sun no longer does 
I heir man pages in raff format. How can people access the man 
page for the stock Sun df command instead of the FSF one? 

A in the case of df , you can actually use the section to differen¬ 
tiate it from the GNU df. For/usr/bin/df it would be: 

man -$ 1m df 

For /usr/ueb/df, it w'ould be: 
man s lb df 

For GNU's df, it would be: 
man -s 1 df 

There are some cases where the third party man pages and the Sun 
man pages will have the same section designation, though. In that 
case T you can specify the path you d like to use on the command line: 

man /usr/share/rcan df 


Q We’ve finally upgraded to the latest and greatest OpenSSH 
(3.6.1p2) on our internal Solaris 8 machines. I can’t seem to 
get password aging to work with the new version, and this is a big 
problem for us. Is there something new that needs to be configured 
in sshd^config or ssh.config that I've just overlooked? (I'm using 
the stock installed ones on the test machine.) 

A Password aging on Solaris 8 is actually broken with 3.6,1. 
See the bug report at: 

http://bugzilla.m1ndrot.org/shoOu9-Cgi7id-24 

There is a third party patch you can apply to 3,6.Ip2 to get it 
working again, though: 

http://www.zip,com,auZ-dtucker/openssh/ 


Q l've been trying to set up sendmaii 8.12.9 to forward all 
“unknown” mail to another host, I've read through the send- 
mail documentation, and I think I have everything right in my me 
file, but when I try to send mail to a nonexistent user, I get user 
unknown error instead of having the mail forwarded onto the other 
machine. Here's my me file: 

divert(Q)dnl 
0STYPE(solaris2)dnl 
FEATURE('nouucp' ( K reject'Jdnl 
FEATURE!'redirect 1 )dnl 
FEATURE( 1 use_cw_fi1e f )dn] 

FEATURES T lccal_prDcina1 U )dnl 
FEATURE!'stickyhost 1 )dnl 
FEATURE!'always.add.donialn'Jdnl 
FEATURE!'access.db')dnl 
LOCAL.USER!'root*Jdnl 
EXPOSED_USER< 'roof Jdnl 
i 

define! 'LtJSER_RELAY\ 'machine, other, doraain'Jdnl 
define! ‘confSAFE.QUEUEL ‘true’) 

define!'eonfPRlVACY_FLAGS f , ‘"authwarnfngs,noexpn ( novrfy' 1 Jdnl 
define! T confTO_JDENT p , 'Os'Jdnl 
define! , eonfSMTP_LGGIN_MSG', 'Jj {NO UCEVJdnl 
HAlLERUocal Jdnl 
NAILER!srotpjdrri 

A lt took some digging for me to find information on this particu¬ 
lar issue, but I believe that your problem is the stickyhost fea¬ 
ture, According to the README, stickyhost is really meant to be 
used in conjunction with the LOCAL_RELAY and MAIL.HUB 
directives. Here's the actual text: 

stickyhost This feature is sometimes used with LOCAL.RELAY, 
although it can be used for a different effect with MAfL.HUB, 

When used without MAIL.HUR. email sent to “user©local.host” 
are marked as “sticky' —— that is. the local addresses aren’t 
matched against UDB. don't go through rule set 5, and are not for¬ 
warded to the LOCAL.RELAY (if defined), 

Wuh MAIL_HUB, mail addressed to “user@Tocal,hosf' is forwarded 
to the mail hub, with the envelope address still remaining 


10 — Sys Admin 


www, sysadminmag.com 


Solaris Administration Supplement 2003 


Here is an example of HTTP Server Single Workload CPU 

Usage; 

HTTP Web Server Workload is using 15% (10 %usr + 5 %sys) 
OVERHEAD is 5% total usage 
Growth Factor is 5 

Total CPU Usage 80% = 5% + (15% * 5) 

The estimated total CPU usage if our workload grew by 5 limes 
would be approximately 80%. 

Here is an example of a Mixed Workload Oracle and HTTP 
W r eb server; 

ORACLE Workload is using 45% (35 %usr + 10 %sys) 

HTTP Web Server Workload is using 15% (10 %usr + 5 %sys) 
OVERHEAD is 15% total usage 
Total Usage = 75% 

Growth Factor for BOTH is 2 

Total CPU usage is 135% = 15% + 05% * 2) + (45% * 2) 

Remember that if you are using a multithreaded application, you 
can think of 100% per CPU. For the workload above, we would 
be running a workload that demands 1.35 CPUs; or if this were a 
single CPU server, you would need to look at purchasing some 
additional hardware. 

The more traditional method for sizing is (Total Sizing = Total 
* Factor). This is generally a less accurate formula because it 
assumes that all resources on the system scale with the same 
factor, w r hich is not generally the case. 


Conclusions 

By using the SE Toolkit and Orca, you will quickly be able 
to establish a visual representation of your Sun servers’ behav¬ 
ior and the applications that are installed. This will enable you 
to see and communicate a clearer and more accurate picture 
regarding the resource usage of all the applications. Finally, 
coupled with some general points, you should be able to 
develop a high-level understanding of the capacity of your sys¬ 
tem based on the current workloads configured on your system. 
With a few workload-based formulas, you can begin your 
capacity planning. 
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the URL specified in your Web server configuration, you should 
see the index page shown in Figure 7. By clicking on the CPU 
usage link at the bottom of the index page, you should get the total 
CPU usage for each process as shown in Figure 8. 

An example of a process with high chld% percentages is our 
system monitoring package, which executes a lot of shell scripts. 
See Figure 9. 

Before we get into the actual planning model and examples, 
you'll need to be familiar with the various performance data val¬ 
ues and how and why they are important. For a quick introduction, 
please see the “Performance Data Values’" sidebar. 


Workload Capacity Planning Model and 
Example 

Now that we have shown how to get the workload data into a 
nice format, lefis finish up with the formula that can be used to 
scale the capacity of a system. 

The formula for workload-based capacity planning should be 
based on the workollator data, because this shows how much of 
the CPU is being used by the workload you are interested in rather 
than a system total. The system total can be considered as over¬ 
head or possible bottlenecks to the workload being scaled. Note 
that the workload shows what the workload uses in (%usr time) 
and what the system does to service the workload in (%sys time). 


Introduction to Performance Data Values 


Wk — This is the assigned numeric workload number ID when 
collecting data. 

Timestamps — Timestamps are in Unix Epoch time format, 
w r hich is the number of seconds after 00:00:00 UTC on 
January 1 1970, 

Command — The workload search filter based on the command 
name 

Args — The workload search filter based on processes arguments. 

User — The workload search filter based on the processes run¬ 
ning user name. 

Cnt — The number of concurrent instances of the running work¬ 
load process. 

usr% — The total percentage of lime the workload spent running 
on the CPU executing in the developer code.^4 

\VS% — The total percentage of time the workload spent running 
on the CPU executing in the kerne! servicing the system calls 
for your application, *4- 

cpwt% — The total percentage of time the workload was ready 
to run, but spent in the RUNQUB watting to get back on the 
CPU to begin executing again. 

chld% — The total percentage of CPU time of the child 
processes that have exited. You will see high chld% percent¬ 
ages if your application is forking off multiple shell scripts. 
This happens typically with monitoring applications, (See 
Figure 9 for a great example, represented as reaped children). 

sizeM — The virtual address space for the application. Most long- 
lived processes grow their address space quickly when they first 
start up then stabilize. If a process’s virtual memory size contin¬ 
ues to increase, then it es quite likely that it has some kind of 
memory leak. This rule is useful for long-term analysis; it does 
not apply recently created processes. 

pf — The number oi page faults that had been triggered. Page 
faults occur when the process returns to Ehe CPU and finds that 
a page in memory no longer exists and has to be reclaimed back 
to the memory tree fist. This can indicate a memory-shortage if 
seen in high numbers. A 

pgwt% — The total percentage of time waiting for page fault to 
be completed. The time spent waiting for page faults for non¬ 
executable data pages indicates I/O activity. The pgwt% can 
be used to indicate a memory shortage 

ulkwt% — The total percentage of CPU time the user-lock from 
idle threads stopped on a semaphore. This is relevant on sys¬ 
tems with databases. 

Process with Blocked Threads Rule — Multithreaded 
processes can consume more than 100% CPU time, and can 


also report more than 100% of wait time. Every thread that is 
blocked on a semaphore will report via the microstate “user 
lock wait time"'. This can be confusing as you could see a 
process accumulating hundreds or thousands of wait% time 
in a report. The fact is that there are a hunch of idle threads; 
no CPU resources are being consumed. The Java runtime 
environment is multithreaded, and it is quite common for 
this to occur with Java programs, 

ioK — The total number of characters in Kb read and written in 
re ad/write system calls. 

sysc — The total number of system calls executed for the work¬ 
load. 

vetx — Voluntary context switches happen when the application 
leaves the CPU on its own. An example would be waiting for 
the system to service a disk I/O. If you see this consistently 
high, then it may indicate that the application is spending 
more time waiting for its own interrupts to be completed than 
processing on the CPU. Voluntary context switches occur 
when the process blocks in a system call to wait for something 
else to complete, 

ictx — Involuntary context switches happen when a process is 
interrupted or kicked off the CPU by a high priority process 
or a system interrupt. High involuntary context switches can 
indicate that the system has lots more processes with higher 
priority keeping your process from getting CPU time. An 
involuntary context switch occurs when another higher pri¬ 
ority process has taken over the CPU, and often occurs 
when the process has used up an entire time slice and had its 
priority reduced as a consequence. So, voluntary context 
switches should be expected, but involuntary context 
switches indicate that there is some contention for Lhe CPU 
and a bottleneck exists. 

msps — The milliseconds per context switch shows how long the 
process ran on average before h switched off the CPU. 1 seconds 
“ 1000 milliseconds and I minute = 60000 milliseconds. 


*The microstare measures of user and system CPU time don't miss anything, 
so if they arc zero the process has definitely nut run in the interval, 

+YOU may see the CPU percentage reach over a 100% on a multiprocessor 
machine: the following can be read lo correctly interpret (100%=1 CPU), 

^Swapping happens to idle processes, so when memory is short you would 
expect to see one process reporting a memory shortage and different 
processes reporting that they have been swapped out to make space. 
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processes running as the cis user Also, the 300 after the workolta- 
tor.se on line 14 represents Lhe number of seconds after which the 
process will wake up and capture Lhe workload data in a daily log 
file, located in the directory as defined by the OUTDrR variable. 

An example of the log file created on the system by the 
workollator.se configuration is shown in Figure l , The file is 
created daily in the directory /opt/orca/workload/ with the 
name of isocissvr-OOl-2002-02-19. 

Collecting Data Centrally Using rsync and SSH 

The workload data from the servers must be collected centrally 
on to the Orca server for processing. In Salt Lake, we used a sim¬ 
ple Perl script using rsync (http://www.samba,org/rsync) and 
SSH to read from a file the IP addresses and hostnames of all the 
servers from which dam needed to be collected. The SSH account 
that was used to authenticate needed only read access to the 
workollator logs in the directory as defined in the OUT DIR vari¬ 
able in ourcis_workoIlator.sh script (see Listing 2). 


Installing Orca 

The next step is to install Orca on a Web server machine. Orca pro- 
vides an excellent ll Gnca installation and configuration guide 11 , located 
in the contents of the package, w hich can be dow nloaded from: 

http://www,orcaware.com/orca. 

Orca needs only to know where the HTML pages are to be stored 
and the location of your Web serv er access log. This information is 
only used for the initial installation and can be changed in an Orca 
configuration tile. The version we are using for our installation is 
orca-0.27b3.tar.gz. 

Introduction to Orca and RRD 

Orca is a toolkit that will graph arbitrary data into HTML doc¬ 
uments using PNG or GIF graphics generated from an RRD 
(Round Robin Database). RRD is the datastore, or database, that is 


Figure 6 Java workload extraction data example 


timestamp 

carcnand 

args 

user count 

usr% 

sys% cpwtft 

chlcft 

Sl2PW 

pf 

pqwt% 

ulkwras 

TDK 

sysc 

VCt* 

Ictx 

msps 

4 1016382482 

java 

* 

cis 

l 

4.0 

1.2 

6 

0 

623 

0 

0 

100 

0 

635 

363 

1 

0.14 

4 1016382452 

java 

Hr 

ds 

2 

3/5 

1.2 

8 

0 

623 

0 

0 

100 

0 

605 

362 

1 

0.14 

4 1016382502 

java 

* 

ds 

2 

4.0 

1.3 

13 

0 

623 

0 

0 

100 

0 

600 

357 

1 

0,15 

4 1016382512 

java 

* 

ds 

2 

4.3 

1.3 

8 

0 

623 

0 

0 

100 

0 

671 

365 

1 

0,15 

4 1016382522 

java 

it 

cis 

2 

4.1 

1.2 

7 

0 

623 

0 

0 

100 

0 

626 

362 

1 

0,15 

4 1016382532 

java 

IV 

ds 

2 

4,0 

1.2 

12 

0 

623 

0 

0 

100 

0 

604 

358 

1 

0,15 

4 1016382542 

java 

* 

ds 

2 

4,0 

1.2 

7 

0 

623 

0 

0 

100 

0 

631 

362 

2 

0,14 

4 1016382552 

java 

* 

ds 

2 

3.9 

1.2 

S 

0 

623 

0 

0 

100 

0 

607 

362 

1 

0,14 


Figure 7 Sample Workload Page from Salt Lake 2002 Server 
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Figure 3 Workolla tor. cfg — CPU usage graph syn- 

tax example 1 
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Performance tuning 
just got easier! 

Understand exactly what’s happening 

SarCheck translates pages of sar and ps output into a 
plain English or HTML report, complete with graphs, 
system tuning ami hardware upgrade recommendations, 

Maintain full control 

SarCheck fully explains each of its recommendations 
providing you with the information you need to make 
intelligent, informed decisions about your system. 

Plan for future growth 

SarCheck T s Capacity Planning feature helps you to plan 
for growth, before slow downs or problems occur. 

Available for most HP-UX, Solaris r AtX and SCO systems . 
Visit us at http:llwww,sarcheck.comi for a FREE evaluation 


Tel +1-603-382-4200 Fax +1-603-382-4247 
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A starting point is to capture accurate system behavior infor¬ 
mation by using microstate accounting tools. This means your 
tools can harvest the performance data from the system without 
becoming one of the performance problems Lhat needs to be inves¬ 
tigated. The SE Toolkit developed by Rich Pettit and Adrian 
Cockcroft is able to accomplish this; not only are the access meth¬ 
ods very lightweight, but the interpreter itself is written such that 
execution speed is very fast, SE Toolkit is available free from: 

http://www.5et.QDl kit.com 

2. To make your system easy to understand, break down the sys¬ 
tem’s activity. 

By breaking down the system into workload sets, you can 
intuitively analyze the machine in terms of categories or "’slices 
of a pie". This categorizing or slicing is called workload analy¬ 
sis. Because every system runs a mixture of workloads, you just 
need to pick out the relevant processes and then aggregate the 
results. The rest is considered overhead of unplanned activ ity. 
The workollator.se class, which is included in the SE Toolkit 
3,3, was written during the Winter 2002 Olympics, to ease the 
collection of accurate data about a defined workload set in a sin¬ 
gle consistent daily log. This allows you to see what each 
process or workload is doing during the given period of time. 

3. “A picture is worth a thousand words", so make the raw data 
intuitive to understand. 

You need to be able to make sense of the data that you are col¬ 
lecting about your workloads. Looking at thousands of lines of 
performance data is not intuitive, even for gurus. Graphing your 
performance data gives you the best visual representation of all 
of those numbers and is much easier to decipher. One tool that 
enables this visualization of your performance data is Ore a, 
which is available from: 

http://www.orcaware.cora 

The Orca utility is written by Blair Zajac and currently is free, 
Grea uses the RRD for plots by Tobias Oetiker, available from: 

http://peopla.eeHethz.ch/-cietiker/webtooTs/rrdtool/ 
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A Case Study on Capacity Planning 

Bv Mark Ehr* Senior Analyst — Enterprise Management Associates 

Note: The following is excerpted from "The Value Proposition for Capacity Planning/' a white paper prepared by Enterprise Management Associates, To view the 
paper in its entirety, visit wwwieamquesncoin/ema 


A contractor to the US Government that provides services to 
many government agencies fand who requested anonymity) 
has a database engine that contains information on hundreds 
of thousands of military personnel and their dependents. Just a few 
years old. the system has become enormously popular, growing from 
two UNIX servers to nearly 200 since it began operations. 

Such rapid growth was good news for the company, but bad news for 
its IT organization* which was scrambling to keep up with the 
skyrocketing demand for server capacity. To have any chance to catch 
up, the company needed automation and planning technology to 
facilitate grow th. The company also needed performance repotting and 
I racking capabilities to ensure good service and debug performance 
problems. 

In 2001. the company's sole server provider. Sun Microsystems, 
recommended Team Quest® Model to the company’s IT department, 
TeamQuest technicians came to the company and installed Model on 


Since its installation, 
TeamQuest® Model has enabled the 
company to save millions of dollars 


six Web servers to collect data and create some sample models for IT 
staff. Once company officials gave the go-ahead, the lull Model 
package was installed and operational within a few days. 

Since its installation, TeamQuest Model has enabled the company to 
save millions of dollars in over-pro visioning costs, according to 
officials associated with the Model deployment. In one instance, 
Model modeling helped the company find a configuration that spread 
the workload across existing Sun servers, reducing the need for new 
servers from 30 to 20 and saving the organization millions of dollars. 

Today, Model is helping the company maintain consistent levels of 
service across its government customer base, which is a crucial 
element to its business. Model enables the company to model server 
performance and establish service level agreements with realistic 
thresholds, making it easier for the organization to meet or beat 
customer expectations, even as its server infrastructure continues to 
evolve. 



EMA Perspective 

In a difficult economy, every IT dollar matters. The days of solving 
performance problems by indiscriminately buying more servers are 
over, and there is an excellent window of opportunity for deploying 
management technology that can optimize server utilization and 
performance. IT organizations are looking for ways to make better use 
of existing server capacity while reducing the need to buy more 
processor power. 


Through capacity planning IT 
organizations can realize major cost 
savings immediately 


At the same time, it is important to note that IT budgets are fight, and 
that most IT organizations are not prepared to purchase new' 
management or planning software unless they are guaranteed a clear 
and fast return on their investment (RQI). Today’s management tools 
must be affordable, easy to deploy, and show strong results in a very 
short period of time. 

Capacity planning tools meet all of these criteria. Through capacity 
planning, IT organizations can realize major cost savings immediately 
by reducing or eliminating the need to over-pro vision their server 
environments. Capacity planning and modeling technology also can 
guide IT through the server consolidation process, and can help set 
realistic goals for serv ice levels that can be maintained over time. 
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